DOCUMENT RESUME 



ED 444 036 



CE 080 626 



TITLE 

INSTITUTION 

PUB DATE 
NOTE 

AVAILABLE FROM 

PUB TYPE 
EDRS PRICE 
DESCRIPTORS 



IDENTIFIERS 



National Assessment of Vocational Education (NAVE) : Overview 
of Evaluation Plan. 

National Assessment of Vocational Education (ED) , 

Washington, DC. 

2000 - 00-00 

28p.; For NAVE Design papers, see CE 080 512-516. 

For full text: 

http : //www. ed. gov/of fices/OUS/eval/NAVE/evaluat ion. html . 
Reports - Descriptive (141) 

MF01/PC02 Plus Postage . 

Academic Achievement; *Compliance (Legal) ; Educational 
Change; *Educational Legislation; Educational Policy; 

Federal Legislation; Outcomes of Education; Postsecondary 
Education; *Program Evaluation; *Research Methodology; 
Secondary Education; Special Needs Students; * Vocational 
Education 

*Carl D Perkins Vocational Technical Educ Act 1998; 

♦National Assessment of Vocational Education 



ABSTRACT 



The Carl D. Perkins Vocational and Technical Education Act 
as amended in 1998 (Perkins III) requires that a national assessment of 
vocational education (NAVE) be conducted. Several key policy issues are 
likely to shape the course of vocational education and of the NAVE: (1) the 

small federal investment in vocational education relative to overall federal 
education spending; (2) the focus of education reforms on improving students’ 
academic achievement; and (3) changes in accountability and funding 
provisions in Perkins III. The NAVE will seek to discover which strategies 
improve the performance of vocational students, what are the pathways by 
which sub-baccalaureate students prepare for careers, and whether the policy 
shift to flexibility and accountability is likely to in^rove program quality 
and student outcomes for regular and special needs students. The NAVE study 
agenda calls for diverse data collection and analysis methods and the study 
of key issues in both secondary and postsecondary vocational education. 
Because NAVE results must be reported, to Congress by July 2002, the study 
will reflect the very early efforts made in response to new provisions in 
Perkins III. (KC) 
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NATIONAL ASSESSMENT OF 
VOCATIONAL EDUCATION (NAVE) 



OVERVIEW OF EVALUATION PLAN 

ABSTRACT 



The recently amended (1998) Carl Perkins Vocational and Technology Education Act (Perkins 
ni) directs the Secretary of Education to complete an “independent evaluation and assessment of 
vocational and technicaJ education programs under this Act,” and to appoint an independent advi- 
sory panel to advise the Department of Education on how to conduct this national assessment of 
vocational education (NAVE). 

Vocational education is a field in transition, prompted by sweeping changes in state and local 
education priorities. Several key policy issues are likely to shape the course of vocational educa- 
tion, and therefore of the NAVE: (1) the small federal investment in vocational education relative 
to overall federal education spending, (2) the focus of education reforms on improving students’ 
academic achievement, and (3) changes in accountability and funding provisions in Perkins HI. 
Given the educational and political context in which Perkins in operates, as well as specific leg- 
islative changes recently enacted, the NAVE will be guided by several main questions: 

What strategies improve the performance of “vocational students” and how does, or can, 
vocational education contribute to improving academic and occupational skills, access to 
postsecondary education, and earnings? 

What are the pathways hy which suh-haccalaureate students prepare for careers, and 
what is the contribution of workforce reform efforts to improving their training? 

Is the policy shift from set-asides and legislative prescription to flexibility and account- 
ability likely to improve program quality and student outcomes? How do special popula- 
tions fare? 




RESEARCH STRATEGIES TO ADDRESS 
MAIN POLICY ISSUES 



A. Participation, Pathways, Outcomes, and impact 

B. Alignment with WiA 



A. Accountability 

B. Funding 



A. Participation, Outcomes, and Impact 

B. Quality 

C. Effectiveness of Promising Strategies 



Postsecondary Occupational Education 



Secondary Vocational Education 



Program Management 
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Addressing the primary vocational 
policy issues requires a set of 
interrelated but distinct studies. 
The overall research agenda calls 
for diverse data collection and 
analysis methods, including: quali- 
tative case studies, national sur- 
veys, examination of existing data- 
bases, and econometric estimation. 
While some broad themes are 
relevant to both secondary and 
postsecondary vocational educa- 
tion, each also has its own key is- 
sues. Finally, given the timeline 
for implementing Perkins HI, and 
the July 2002 date for reporting 
NAVE results to Congress, the 
studies conducted will reflect the 
very early efforts made in re- 
sponse to specific new provisions 
in Perkins III. 





NATIONAL ASSESSMENT OF 
VOCATIONAL EDUCATION (NAVE) 

OVERVIEW OF EVALUATION PLAN 



I. INTRODUCTION 

Federal support for vocational education, and for understanding its consequences, has a long his- 
tory. As was true with previous vocational legislation, the recently amended (1998) Carl Perkins 
Vocational and Technology Education Act (Perkins HI), directs the Secretary of Education to 
complete an “independent evaluation and assessment of vocational and technical education pro- 
grams under this Act,” in this case, by 2002. As currently designed, this national assessment plan 
will include multiple components. This paper discusses the main issues that NAVE will address, 
and the research strategies that will be undertaken to address those issues. 

Perkins HI (section 114) directs the NAVE to address a wide range of topics, including: (1) im- 
plementation of state and local programs; (2) impact of changes in federal funding formulas; 
(3) teacher quality and teacher supply and demand; (4) student participation in vocational educa- 
tion, (5) academic and employment outcomes; (6) employer involvement and satisfaction with 
vocation education programs; (7) education technology and distance learning; and (8) the impact 
of accountability requirements on program performance. Perkins HI also directs the Secretary to 
appoint an Independent Advisory Panel to provide advice on conducting the NAVE and to submit 
to Congress its own independent analysis of NAVE findings and recommendations (see Appendix 
A for list of panel members). 

Congress has mandated a final NAVE report in 2002 to provide it with information that can guide 
reauthorization of the Perkins Act in 2003. Of primary interest is how some of the new provisions 
in the law have been implemented. Unfortunately, many of the important changes in accountabil- 
ity, integration with workforce development, and increased flexibility will have barely begun. 
Most states have opted for “transitional plans” which in effect, defer implementation of Perkins 
ni provisions until October 2000 (Figure 1). Given the July 2002 date for reporting NAVE 
results to Congress, data collected during the 2000/2001 school year will reflect the very early 
efforts made in response to specific new provisions in Perkins HI. Conclusions regarding the 
longer-term prospects for Perkins HI will require further research. 
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At the same time, however, Perkins HI continues to emphasize several major themes reflected in 
the 1990 Perkins amendments. These include the integration of academic and vocational educa- 
tion, broadening the focus of vocational education to emphasize industries and careers, and 
strengthening the links between secondary and postsecondary education through tech-prep and 
other strategies. The current NAVE will be able to track the extent to which these longer run 
themes are reflected in school practice, and where possible, what effects they have on outcomes 
for students. 

All of these objectives will be addressed in a series of studies soon to get underway. Two factors 
are guiding the design of the NAVE agenda: (1) the current policy environment, and (2) the spe- 
cific research questions that reflect policymaker and practitioner interests. 



A. Policy Context FOR THE NAVE 

Over the past two decades, three prior national assessments of vocational education have been 
conducted.’ Each study sought to focus upon what were thought to be the major issues facing vo- 
cational education at the time — funding, special populations, and economic competitiveness of 
the workforce. The current NAVE must also be sensitive to the dominant educational issues of its 
time. 

Vocational education is a field in transition, prompted by sweeping changes in state and local 
education priorities. New goals, program offerings, and terminology increasingly characterize 
vocational education. Federal legislation has encouraged several major changes — from an historic 
emphasis on entry-level job preparation in semi-skilled occupations to a broader focus on prepa- 
ration for careers that offer high wages and requires higher level skills; from preparing students to 
enter the workforce directly after high school to providing students with the choice of pursuing 
employment or attending college, or as is increasingly the case, doing both simultaneously; and 
from expecting vocational students to do less well in school than other students, to holding such 
students to the same academic standards as others. Many of the overarching issues that NAVE 
will address will consider whether this transition in the field of vocational education is “on track.” 

Several key policy concerns are likely to shape this assessment: 

• Federal Funding For Vocational Education. Federal budgets are widely regarded as 
a basic indicator of policy priorities. Although overall funding on Department of 
Education (ED) programs has increased by 177 percent from FY 1980-FY 1999, vo- 
cational education funding increased by only 47 percent during the same period (Fig- 
ure 2).^ In 1980, funding for vocational education was about 6 percent of total ED 
expenditures; it has now shrunk to about 3 percent. 



'1982, 1989, and 1994. 

^Figure 2 shows the trends in federal funding for vocational education over the past 20 years, and compares funding for 
vocational education with overall funding for Department of Education programs. 
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Figure 2 

FUNDING TRENDS IN EDUCATION PROGRAMS 
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These trends strongly suggest that for nearly two decades vocational education has 
been increasingly viewed as less worthy of investment than other areas. One reason 
for this pattern may be that vocational education has historically prepared students 
for work at a time when priorities have shifted toward academic preparation for col- 
lege. To the extent that vocational education has changed to address these as well as 
more traditional objectives, it will be important for NAVE to describe and assess this 
development. 



Although overall federal funding 
for vocational education has not 
changed substantially, the Perkins 
Act remains the largest single 
source of ED funds directed to 
high schools. Comparing ED’s 
three major sources of funding 
for high schools (only a share of 
the funds from each program is 
directed at high schools), Perkins 
Act funds spent to help high 
school students are more than 
Title I and TRIO funds combined 
(Figure 3).^ 



Figure 3 

SELECTED EDUCATION DEPARTMENT 
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^Figure 3 compares ED funding of high schools from three major sources — the Perkins Act, Title I of Improving 
America’s Schools Act (lASA), and the TRIO program. 
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Given the major role that Perkins funding plays at the high school level relative to 
other ED programs, it makes sense to think about how this investment contributes to 
high school improvement generally, and the contribution that vocational education 
makes, or could make, to such efforts. 

• Academic Reform. There is little question but that the principal focus of recent edu- 
cation reforms has been on improving students’ academic achievement and increas- 
ing their opportunities to attend college. Federal vocational education policy now 
places top priority upon ensuring that vocational education students are academically 
well prepared for both careers and/or success in postsecondary education. But what 
is, or ought to be, the responsibility of vocational educators for ensuring that students 
leave high school with both a solid academic foundation as well as technical skills? 
One major goal of NAVE should be to help Congress better understand what the 
contribution of vocational education is to achieving these objectives and how this 
contribution can be increased. 

• Changes in Perkins III. Although Congress did not alter the basic structure of the 
Perkins Act in the 1998 reauthorization, it did make several important substantive 
changes. Among these are: (1) increased emphasis on academics; (2) greater flexibil- 
ity in the use of funds through elimination of major set-asides for gender equity and 
other rules governing the use of funds; (3) a higher proportion of funds directed to lo- 
cal programs and the establishment of a 10 percent reserve fund; and (4) creation of a 
“higher stakes” accountability system. At the same time, the Workforce Investment 
Act of 1998 (WIA) also encouraged greater integration of vocational education and 
the workforce development system. Implementing these changes will be a major 
challenge for school districts and postsecondary institutions. 



B. Main Policy issues 

Given the educational and political context in which Perkins HI operates, as well as specific leg- 
islative changes recently enacted, the NAVE will be guided by several main questions: 

1. What strategies improve the performance of “vocational students” and how does, or 
can, vocational education contribute to improving academic and occupational skills, ac- 
cess to postsecondary education, and earnings? 

Standards-based reform is fundamentally altering policies and practices from kindergar- 
ten through high school. NAVE will examine the role of standards — both academic and 
occupational — in vocational education, the effects on students’ academic and technical 
preparation of promising reform strategies, and the role of federal policy, as implemented 
by states and communities, in facilitating implementation of effective programs of study. 

2. What are the pathways by which sub-baccalaureate students prepare for careers, and 
what is the contribution of workforce reform efforts to improving their training? 

In passing Perkins HI and the WIA, Congress was concerned that federally supported job 
training programs operated independently of vocational education. Congress expects that 
linking Perkins HI and WIA together will encourage better integration of vocational edu- 
cation and workforce development policies. Issues to be examined include: (1) the role of 
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postsecondary vocational in developing state workforce training plans; (2) the importance 
of unified planning; (3) the early impact on postsecondary institutions (e.g., programs of 
study, one-stop career centers, accountability) of efforts to align Perkins III and 
workforce development, and (4) patterns of enrollment and participation in postsecondary 
occupational education, and their relationship to outcomes and impact. 

3. Is the policy shift from set-asides and legislative prescription to flexibility and account- 
ability likely to improve program quality and student outcomes? How do special popu- 
lations fare? 

For the past two decades, federal policy has focused on serving those most at-risk, com- 
monly termed the “special populations.” Perkins III represents a major shift in direc- 
tion — eliminating set-asides for “single parents...,” requirements that local funds be pri- 
oritized to serve the highest concentrations of special populations, and requirements to 
coordinate with the Individuals for Disabilities Education Act. In its place is an increased 
emphasis on accountability, including the requirement that states track the progress of 
special population groups. Has increased flexibility resulted in changes in educational 
priorities or practices? Have “at-risk” populations been helped or hurt as a result? Are ac- 
countability requirements improving the quality of vocational education for all students? 



II. NAVE RESEARCH AGENDA 



;S:# 



Figure 4 

RESEARCH STRATEGIES TO ADDRESS 
MAIN POLICY ISSUES 



■ ■ 



Addressing the primary vocational 
policy issues requires a set of inter- 
related but distinct studies. No one 
study could encompass the broad 
array of interests and questions. 
Moreover, the overall research 
agenda calls for diverse data col- 
lection and analysis methods, in- 
cluding: qualitative case studies, 
national surveys, examination of 
existing databases, and sophisti- 
cated econometric estimation. Fi- 
nally, while some broad themes are 
relevant to both secondary and 
postsecondary vocational educa- 
tion, each also has its own key 
issues. 

For these reasons, the NAVE plans 
to undertake studies in seven main 
areas (Figure 4). These are described below, grouped together by major topic: secondary voca- 
tional education, postsecondary vocational education, and program management. 



Secondary Vocational Education 

A. Participation, Outcomes, and Impact 

B. Quality 

C. Effectiveness of Promising Strategies 

Postsecondary Occupational Education 

A PartlrJnatinn Pathu/avs niifnnmpa anri Imnant 




NAVE Overviews 



SECONDARY VOCATIONAL EDUCATION 



The role of vocational education at the secondary level is evolving. Many policymakers, educa- 
tors, and parents remain concerned about whether secondary vocational programs serve students’ 
best interests in the long run. Previous studies suggest that completion of a vocational program 
has a positive effect on employment, at least in the short run, if students do not pursue postsecon- 
dary education'* * and if they obtain training-related jobs.’ However, the earlier research also indi- 
cates that vocational education contributes to neither academic achievement nor postsecondary 
enrollment — the path desired by most students and their parents. Studies undertaken by the last 
NAVE confirmed that vocational instruction in the early 1990s was still largely traditional, with 
little focus on academic skills.® These concerns about the quality and outcomes of vocational 
education have contributed to the decline in participation of high school students since the 1980s. 
These same concerns have raised the stakes for gaining a clearer understanding of: (1) how par- 
ticipation in and impacts of secondary vocational education have changed since the earlier stud- 
ies, (2) the extent to which secondary vocational education now reflects high quality practices, 
and (3) whether promising career education reform strategies are proving to be effective. 



A. PARTICIPATION AND OUTCOMES 

Perhaps the most important issue for vocational education, and for the current NAVE, is who 
participates at the secondary level and how well they fare in school and beyond. At least through 
the early 1990s, vocational education had come to be stigmatized as a high school “track” for stu- 
dents with poor academic capabilities, special needs, or behavioral problems.’ This result alone 
would not be problematic, but vocational programs also appeared to contribute little “value 
added” to student outcomes. In the current climate, with increasing federal emphasis on program 
performance and on high academic achievement for all students, vocational education has much 
to prove. 

Over the last five years, however, career education has been given new prominence by several 
initiatives targeted to students other than those traditionally served by vocational programs. In 
addition, some states and districts have worked to strengthen vocational courses.® Whether any of 
these efforts have successfully broadened the appeal and improved the impact of occupational 
programs needs careful examination. 



‘‘Ferran Mane, Trends in the Payoff to Academic and Occupational-Specific Skills; The Short and Medium Run Returns 
to Academic and Vocational High School Courses for Non-College Bound Students (Working paper # 98-07) (Ithaca, 
NY: Center for Advanced Human Resource Studies, Cornell University). 

^National Assessment of Vocational Education, Final Report to Congress, Volume I, July 1994. 

‘Ibid. 

’ibid. 

*Alan Hershey, Marsha Silverberg, Joshua Haimson, Paula Hudis, and Russell Jackson, Expanding Options for Stu- 
dents: Report to Congress on the National Evaluation of School-to-Work Implementation (Princeton, NJ: Mathematica 
Policy Research, October 1998). 
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Figure 5 

SECONDARY VOCATIONAL PARTICIPATION 
AND IMPACTS 

Research Issues Study Plans 

• Enrollment Trends 

• Special Populations Participation 

• Course Taking 

• Academic Achievement 

• Postsecondary Enrollment 

• Labor Market Benefits 





• NAEP Transcript Study 

• Analyses using NELS, HS&B, 
NLSY, and NAEP 

• NELS and HS&B Academic 
Achievement Comparison 

• Analyses of state data 




Who participates in secondary vocational education? 

Updating the enrollment trends and characteristics of students served by high school-level voca- 
tional programs is of interest, particularly in light of growing reports from state directors that en- 
rollment patterns have been changing. Congress and most educators, remain committed to pro- 
viding students from special populations access to vocational education. However, many believe 
the quality of vocational programs is unlikely to improve without attracting a broader segment of 
the student population; or that the participation of a more diverse set of students will signal that 
quality improvements are being made. Several questions must be addressed: 

■ Who enrolls in secondary vocational education and to what extent has this picture 
changed over the 1990s? How do the characteristics of vocational students compare 
with those of nonvocational students? 

■ Does enrollment vary by high school setting and vocational program area? 

■ Has the course taking (academic and technical) of vocational students changed? 



PLANNED STUDIES 

Descriptive statistics from national data. NAEP transcript data (1998) provide the 
most recent evidence of secondary vocational participation nationwide. Using NAEP 
and student transcripts from other data sources, trends in student characteristics, 
course taking and achievement will be documented. 
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What is the impact of vocational education at the secondary level? 



As a result of changing objectives and educational priorities, it is important to understand the 
contribution of secondary vocational education to traditional objectives (technical competency, 
labor market outcomes, and general employability skills) but also to academic achievement and 
postsecondary enrollment. Among the research issues are 



■ To what extent does vocational education contribute to students’ academic achieve- 
ment and chances of attending and succeeding in college? 

■ What effect does secondary vocational education have on students’ technical prepa- 
ration for work, employability skills, and ability to meet employers’ expectations? 

■ What is the impact on wages and earnings? Li the short run? In the longer run? For 
noncollege-bound students? College-bound students? 

■ How do these outcomes vary for different groups of students, particularly students 
from “special populations?” 



PLANNED STUDIES 

Analyses of national and state databases: NAVE will use several national databases 
(HS&B, NELS, NLSY, and NAEP) to (1) document trends in student achievement, earnings 
and other outcomes; and (2) assess the ejfects of vocational education (on average and for 
different programs of study and student populations). 

NAVE will also explore possible use of state level data systems to provide information on 
students’ technical competencies and employability skills, as well as employer satisfaction 
with vocational students. 



B. Quality of Secondary School Vocational Education 

For nearly a decade, federal policy has attempted to improve the quality of vocational programs 
by strengthening the connection between vocational education and mainstream educational ob- 
jectives at the high school level. These vocational improvements are intended to keep pace with 
and complement other reform efforts in high schools. States and local districts have been raising 
the academic coursework and skills required for graduation, making high academic achievement 
the paramount marker of a school’s success. While other measures of school performance are also 
important (e.g., placement into higher education or career-oriented employment, reductions in 
drop-out rates, technical competency), efforts to increase academic attainment are likely to con- 
tinue as a focus for school improvement. A major policy issue facing vocational education, then, 
is how it can support this central mission for high schools. 

The “quality” of vocational education is clearly critical to this objective. Perkins in builds on 
prior legislation in emphasizing program improvement. While it may be too early to fully judge 
the educational system’s response to Perkins HI, Congress will certainly be interested in the 
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extent to which actual practice is consistent with legislative and other views of what constitutes 
“effective” vocational programs. 



How can schools improve the academic performance of vocational students and what, 
if anything, is the relationship between vocational education and those improvement 
efforts? 

Federal vocational education policy now places priority on ensuring that secondary vocational 
education students are academically well prepared for careers and success in postsecondary edu- 
cation. For some districts and schools, meeting these objectives requires substantial changes, in- 
cluding new policies or requirements, shifts in instructional methods, or modifications to course 
content; some schools have already undertaken these reforms. While Perkins HI provides guid- 
ance on program improvement strategies, identifying the approaches used by schools that have 
actually raised the academic (and technical) competence of vocational students will be of great 
benefit to both policymakers and practitioners. Several critical questions need to be addressed, 
such as 

■ Do high schools that have succeeded in improving vocational students’ achievement 
organize vocational education differently than do schools that have been less suc- 
cessful? 

■ What vocational education practices and approaches appear most promising in pro- 
moting academic achievement and technical competence? 

■ What is the relationship between vocational education improvement and school re- 
forms underway in many states and local communities? 

■ Does federal vocational education policy, as implemented by states and districts, fa- 
cilitate or impede implementation of effective programs of study at the secondary 
level? 
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PLANNED STUDY 



Comparative Case Studies of Schools, Districts, and States: In-depth case studies will docu- 
ment and contrast state and local policy and school practice, and how they might contribute 
to student outcomes. Through intensive site visits, the case studies will obtain descriptive in- 
formation about the extent and quality of vocational programs, state and district efforts to 
support and improve the programs, and the linkages between these efforts and schools reform 
and workforce development initiatives. Overall employer satisfaction with graduates of voca- 
tional programs will also be gauged with data from an ongoing National Employer Survey. 



^ What is the quality of vocational education and to what extent are federal strategies for 
improving vocational education quality reflected in actual classroom practice? 

Perkins HI continues to emphasize several basic reform strategies, the genesis of which was in 
Perkins II; e.g., integrating academic and vocational education, linking secondary and postsecon- 
dary vocational programs, and broadening vocational curriculum beyond its traditional emphasis 
on entry-level job preparation. It is therefore important to examine whether nearly a decade of 
federal efforts to improve the quality of programs have found their way into teaching approaches 
and classroom organization. This examination will be guided by several research questions, such 
as 



■ Are states and communities making progress in implementing key reform strategies 
emphasized in Perkins HI? What other approaches reflect quality in vocational edu- 
cation? 

■ To what extent has professional development been used to support these strategies? 
What kind of preparation and qualifications do vocational teachers have that will 
help them make these improvements? 

■ How does the quality of vocational instruction and teaching vary among different 
institutions (comprehensive high schools, vocational high schools, and area voca- 
tional schools), different communities, or the demographic characteristics of stu- 
dents? What factors affect the depth or quality of vocational education? 

■ What are the barriers to improving the quality of vocational practice? 
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PLANNED STUDIES 



Several approaches will be used to measure the prevalence of vocational improvement 
strategies. 

Teacher Survey and Other Teacher Data: In the year 2000, the NAVE will survey a nation- 
ally representative sample of vocational teachers to collect information on curriculum and 
instructional methods. Information on teacher preparation and qualifications will come from 
the Schools and Staffing Survey (SASS) and from an analysis of PRAXIS test score results 
(used for teacher certification in many states). 

Student Survey: Analyses of existing student surveys will help identify the frequency of cer- 
tain practices and the extent to which they are becoming more common for vocational and 
other students: (1 ) survey of seniors in eight states in 1996, 1998, and 2000 (Mathematica), 
(2) National Longitudinal Survey of Youth-97. 

Case Studies: Comparative case studies will provide data on the quality of practice, barriers 
to improvement, and use of professional development. A significant part of the case studies 
will be a focused examination of selected vocational teacher instruction, including classroom 
observation and a review of lesson plans, competency tests, planned projects, and student 
work as concrete evidence of instructional practice. 



C. Effectiveness of Promising Vocational Education Strategies 

In recent years, a variety of programs and reforms have been introduced to improve the outcomes 
of secondary vocational students. Some approaches have been narrowly conceived (making minor 
adjustments to a small set of courses or activities) while others involve broader curriculum and 
institutional changes. Many of these reform models and practices have become popular, expand- 
ing to more schools or becoming institutionalized in state policy. Some have been supported with 
federal grants. 



Until recently, however, few studies have been conducted of sufficient size and rigor to provide 
evidence on how well and why these reforms work. Schools have to a large extent relied on lim- 
ited information that such interventions as Tech-Prep, career academies, career clusters, and High 
Schools that Work are effective in improving student performance. Now that these and other re- 
forms have been in place for some time, it is reasonable for policymakers and educators to seek 
answers to the following questions: 
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Figure 7 
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Do these strategies, when well implemented, improve student outcomes? 

Although developers and advocates of particular reform models have long cited student success, 
in fact there is little concrete evidence of the programs’ positive impacts. Studies found that im- 
plementation of these initiatives has been uneven, which has in some cases stymied efforts to 
conduct large-scale evaluations of effectiveness. With longer experience, however, examples of 
outstanding implementation have emerged that can be used as test cases. Of particular interest are 
comprehensive program models that can be applied to either vocational students or as part of 
whole-school reform. Research on these models should address the following questions: 

■ To what extent do these promising interventions make a difference in key student 
outcomes: academic achievement, technical competence, postsecondary enrollment, 
or employment? 

■ What other outcomes are associated with the new strategies (e.g., technical literacy, 
work readiness, employer satisfaction)? 

■ Are there differences in impacts for different groups of students (defined, for exam- 
ple, by demographics, socioeconomic status, skill level, or occupational interest)? 



What specific elements do approaches that seem to work have in common ? 

While knowing which programs or reforms make a difference in student outcomes is important, 
so is understanding the elements that make some initiatives successful. For policy and technical 
assistance purposes, it is critical to determine whether key outcomes result primarily from, for 
example, innovative instructional techniques, changes in administrative structures, or simply 
raising core requirements. Looking across all of the studies may provide answers to the following 
key questions: 

■ Which features of reform are essential to producing positive outcomes? 

■ Do the importance of these features vary across school settings? 
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PLANNED STUDIES 

Evaluations of Promising Programs and Practices: The NAVE will draw primarily on studies 
already underway to assess the impacts of promising interventions. These studies may include 
but not be limited to evaluations of 

Career Academies: a random assignment evaluation conducted by MDRC; 

High Schools That Work: a longitudinal and comparative study of school-wide re- 
form being conducted by MPR Associates; 

Tech-Prep: a matched comparison group study conducted by NCRVE; 

Career Magnet High Schools: possible extension of this random assignment evalua- 
tion to assess postsecondary and employment outcomes; and 

Other: NAVE may commission shorter investigations of particular, innovative state or 
local practices that have the potential to improve vocational student outcomes ( e.g., 
technology education curriculum, high stakes end-of-course exams in North Carolina 
and Oklahoma that measure student competencies, etc.). 



NAVE Overview— 13 



POSTSECONDARY OCCUPATIONAL EDUCATION 



Vocational education is both a high school and postsecondary program, although Perkins HI, as 
written, seems primarily focused on high schools. Research has shown that there are significant 
differences between students in high school and in college with respect to the reasons they enroll 
in vocational courses as well as in overall enrollment trends. Moreover, the primary institutions 
that deliver vocational training — high schools and community colleges — are fundamentally dif- 
ferent. Nonetheless, Perkins HI makes few distinctions between high schools and colleges with 
respect to federal objectives, program strategies, or accountability requirements. 

Two issues are quite distinctive about postsecondary occupational education and will receive sub- 
stantial attention under the NAVE. Growing evidence on participation and outcomes suggests that 
vocational education at this level is beneficial. Postsecondary vocational education also has direct 
implications for workforce development. 



A, Participation and Outcomes 

In contrast to vocational education at the secondary level, occupational program enrollments at 
the postsecondary level continue to grow at a rate in keeping with the sub-baccalaureate level 
more generally. In addition, available evidence clearly establishes a shift toward enrollment of 
older students. Such a shift in the demographics potentially signals a diversity of roles if not a 
changing role for occupational education at the postsecondary level — which may have implica- 
tions for federal policy. 
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The economic benefits of postsecondary occupational education are a key measure of the impor- 
tance and impact of occupational programs. Most analyses in the literature find a significant re- 
turn to an associate’s degree. However, there are two caveats: first, there is evidence to suggest 
that the benefits are uneven;® and second, very few students complete an associate’s degree. 
NAVE will examine both the participation and benefits of postsecondary occupational education. 
In addition to the benefits for those who complete a degree, NAVE will examine the benefits of 
occupational education by field, for those who do not earn degrees, and for particular sub-groups 
of students. 

Who participates in postsecondary occupational education and why? 

Knowing the characteristics of those who enroll, and how enrollments are changing provides an 
important context for studying postsecondary occupational education. Of growing importance is 
an older student population that pursues more nontraditional pathways. The key questions to be 
addressed are: 

■ Who enrolls in postsecondary occupational education and how has this pattern 
changed? How do the characteristics of those who pursue occupational education 
compare with those in nonoccupational programs at the sub-baccalaureate level? 

■ In what types of institutions do postsecondary occupational students enroll? 

■ What are the occupational enrollments by program area? Have enrollments by pro- 
gram area changed over time? Are patterns of participation in nontraditional pro- 
grams changing? How important is “not-for-credit” course taking? 

■ Has there been a shift in the timing of when postsecondary occupational students en- 
roll? 

■ What are the educational goals of postsecondary occupational students? Does “rea- 
son for enrollment” vary by student characteristics? 



PLANNED STUDIES 

Descriptive Statistics from National Data: Enrollment patterns and trends by student 
characteristics will be presented using NPSAS (for credit course taking), NHES (inclusive 
of not-for-credit course taking), and Vocational Education in the United States: Toward 
the Year 2000. It may be possible to also use a survey of community college students 
planned by AACC-ACT to present evidence on student characteristics and stated goals for 
a broader student group (inclusive of those who take not-for-credit courses). Emphasis on 
special populations participation will be highlighted. 



’Not all fields of study lead to significant positive rates of return. See W. Norton Grubb, Learning and Earning in the 
Middle: The Economic Benefits of Sub-baccalaureate Education (New York: Teachers College). 

'^ased on the HS&B sophomore cohort, of all students who enroll in a 2-year college, about 16 percent attain an asso- 
ciate’s degree. In contrast, nearly 60 percent of those who enroll in a 4-year college complete at least a bachelor’s de- 
gree. Thomas J. Kane and Cecilia Elena Rouse, “The Community College: Educating Students at the Margin Between 
College and Work,” Journal of Economic Perspectives (Winter 1999): 63-84. 
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What is the contribution of occupational education at the postsecondary level? 

The increasing prevalence of degree noncompletion suggests the need to both understand the 
trends in and to more fully document the contribution of postsecondary occupational education to 
wages and earnings beyond simply looking at those who attain a degree. 

■ How important is degree completion compared with skill acquisition (i.e., are alter- 
native measures, such as industry-generated credentials, more beneficial in the labor 
market than traditional measures?) Are degrees less beneficial and thus less impor- 
tant in some fields? 

■ What is the contribution of postsecondary occupational education to wages and 
earnings? 

■ Does the impact vary by student characteristics, course taking, or field of study? 



PLANNED STUDIES 

Analyses of National (and State) Databases: Several data sets will be used to measure the 
benefits of postsecondary occupational education. 

Rate of Return Studies: Short and longer-term rates of returns will be calculated for 
both degree completers and noncompleters using national data sets such as HS&B, 
NELS, and BPS. The “value added” contribution of occupational programs will be 
calculated using HS&B. SIPP (and possibly state data) will be used to calculate rates 
of return by field of study. 

Studies of Other Benefits: NAVE will potentially make use of state-level data for in- 
formation on technical competencies of, employability skills of, and employer satis- 
faction with postsecondary occupational students. 



B . Postsecondary Occupational Education and Its Alignment with WIA 

A major issue at the postsecondary level is coordinating occupational programs with the 
workforce development system. When Congress enacted Perkins III and the WIA, it believed that 
a plethora of job training programs created excessive administrative burden upon states and dis- 
couraged access to services. Therefore, Congress is likely to be particularly interested in the rela- 
tionship between Perkins and the WIA. 

Workforce development is likely to be affected not only by the institutional alignment of Perkins 
and WIA, but also by the extent to which students complete their training. Although there are still 
many occupations that require only minimal training and lower-level education, occupations 
requiring at least an associate’s degree are projected to grow faster and with larger projected 
numerical increases than those requiring less education or training." Despite (or perhaps as a 
consequence of) increased enrollments in postsecondary education, degree completion rates are 



"George T. Silvestri, “Occupational Employment Projections to 2006,” Monthly Labor Review (November 1997): 
58-83. 
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low and are decreasing at the sub-baccalaureate level. Of particular concern is the dispropor- 
tionately high and growing rate of postsecondary dropping out among “disadvantaged students” 
(such as students of lower socioeconomic status, minority students, and those with lower high 
school grades). Given the diversity of the students who attend postsecondary institutions, the 
explanation for the comparatively low completion rates is unclear and more than likely many- 
layered. The NAVE intends to shed light on this issue by better understanding educational path- 
ways and their implications for workforce development. 
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^ What role does postsecondary occupational education play in development of state and 
local workforce development strategies? 

The 1998 WIA presents opportunities to integrate occupational education with workforce invest- 
ment systems (through the optional use of state-unified plans of WIA and Perkins and the provi- 
sion of workforce development services, through one-stop career centers, at postsecondary 
providers). Several basic questions need to be addressed: 

■ What is the relationship between local workforce boards and postsecondary occupa- 
tional programs? To what extent do local workforce boards impact postsecondary 
occupational programs? 

■ What is the interaction/relationship between postsecondary institutions and one- 
stops? Are one-stops used to integrate postsecondary occupational education and 
other workforce training programs? 



'^D. Boesel and E. Fredland, Is There Too Much Emphasis on Getting a College Degree? (Washington, DC: National 
Library of Education, June 1998). 
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PLANNED STUDY 



Case Studies of Postsecondary Institutions : In-depth case studies will be undertaken 
to examine the role played by postsecondary institutions in development of a coherent 
state or local approach to worlrforce development. A major objective will be to ex- 
amine the early response of community colleges (primarily) to the WIA. The study will 
focus on the implications that efforts to integrate workforce development strategies 
have on occupational programs and institutional relations with area business. 



What is the quality of postsecondary occupational programs (as measured by available 

indicators)? 

Several dimensions of program quality are the: (1) nature of programs offered (as measured by 
faculty quality and responsiveness to labor market demand); (2) “the program of study” students 
actually pursue;*^ and (3) results (as measured by employer satisfaction). Relevant research ques- 
tions include 

■ What are the qualifications of occupational faculty (compared to past/academic 2-yr. 
faculty)? To what extent are postsecondary occupational faculty involved in profes- 
sional development activities and of what type? 

■ Are the occupational programs offered related to local labor market needs, and do 
they change adequately in response to local labor market needs (e.g., do enrollment 
trends coincide with anticipated labor market demand)? 

■ What is the projected level of education needed for mid-level occupations, and do 
students attain the level of education consistent with projected labor market re- 
quirements? Do educational requirements vary by occupational field or by geo- 
graphical region? 

■ Are employers satisfied with postsecondary occupational education? To what extent 
are they involved in postsecondary occupational programs (e.g., prevalence of co- 
ops, involvement on advisory boards, use of postsecondary institutions in hiring 
practices etc.)? Are students satisfied with postsecondary occupational education? 

■ Now that states are no longer required to set aside funds for a sex equity coordinator 
or special populations, how are these issues addressed? 

■ What is the typical postsecondary occupational education pathway? How efficiently 
do occupational students progress toward goal completion (i.e., are course-taking 
patterns coherent and/or of a meaningful quantity or are students simply “milling 
around”)? What are the persistence patterns of occupational students? 

■ As measured by degree/certificate attainment or critical number of courses, are 
completion rates of occupational students commensurate with those of non- 
occupational students at the sub-baccalaureate level? Do completion rates vary by 
student characteristics (old versus young; by goal)? 



is, what courses are students taking or combining, and what is their progress? 
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■ What are the factors that contribute to longer persistence and higher completion 
rates? How is persistence and completion affected by differences in student goals 
and pathways? 



PLANNED STUDiES 

Analyses of National (and State) Databases. A number of analyses using existing data 
sets will explore the quality of postsecondary occupational education. 

Faculty Characteristics: Using the National Survey of Postsecondary Faculty, 
descriptive statistics of faculty qualifications and professional development 
activities for occupational faculty will be documented (in comparison to other 
2 -year faculty). 

Labor Market Trends: The NAVE will make use ofIPEDS and BLS employment 
data to track the correspondence between fields of enrollment and degree com- 
pletion and labor market trends. 

Satisfaction Measures: NELS, BPS, National Employer Survey, and (possibly) 
state data all solicit information about either student or employer satisfaction 
with occupational training. 

Studies of Educational Pathways: Analyses (using BPS, NELS, HS&B, and 
B&B) will investigate such issues as persistence, student pathways through post- 
secondary occupational education including efficiency of course taking, and 
completion. 

Case Studies: Interviews with state and college officials will examine Perkins III 
changes, such as elimination of set-asides and the requirement for a sex equity 
coordinator. 




22 
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PROGRAM MANAGEMENT 



Federal vocational legislation has for some time used funding and accountability provisions to 
achieve specific policy goals. Perkins HI is no exception. Perhaps the most significant changes in 
the new legislation involve these program management tools. Understanding how the new provi- 
sions are implemented, and the consequences of them, will be an important issue for Congress 
and for the NAVE. 



A. Accountability 

Increased accountability is a major component of Perkins HI. While states have been expected to 
gather information on student outcomes since 1990, few states have been able to consistently and 
systematically do so. The 1998 Act raises the requirements for state reporting of student outcome 
and the potential rewards and consequences for states that can and cannot do so. 

Perkins HI requires each state to develop a system of measurements and to establish expected 
levels of performance in four categories: 

1. Student attainment of academic, vocational, and technical skill proficiencies. 

2. Completion of a secondary or postsecondary degree or credential. 

3. Placement and retention in postsecondary education, advanced training, employment or 
the military. 

4. Participation and completion of programs that lead to nontraditional employment. 

In addition, each state is required to report on the progress of special populations with respect to 
each of these categories. 
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The major changes in Perkins III accountability have less to do with the types of measures that 
are required than they do with how performance data, once collected, is to be used. Perkins III 
requires states to negotiate quantifiable levels of performance''* with the Department of Education 
and to report yearly to the Secretary on progress made in meeting these standards. Moreover, the 
Secretary is to make this information available publicly and to compile state-by state compari- 
sons. Finally, Perkins III raises the potential stakes associated with performance. Failure to meet 
state-level performance standards could eventually result in loss of Perkins funds. As a reward, 
however, section 503 of the WIA provides incentive grants to states that exceed performance lev- 
els under the Perkins Act, Adult Education and Family Literacy Act, and WIA Title I. 

Given the significance of these provisions, NAVE intends to address two main issues regarding 
accountability: 

^ Have states, school systems and colleges developed appropriate measures, standards, and 

data-gathering systems? 

Studies undertaken in the early 1990s of initial efforts to implement Perkins II accountability re- 
quirements raised an important concern that remains relevant today. States were generally able to 
meet (or exceed) legislative requirements, but the quality of their systems was uneven. A more 
recent article suggests that “states have made dramatic progress toward meaningful accountability 
systems,” perhaps spurred by the realization that accountability is not a passing fad.'^ Anecdotal 
information indicates that Perkins III accountability requirements have been taken much more 
seriously by federal and state officials than was the case under the previous Act. However, sev- 
eral critical questions need to be addressed: 

■ Are data systems in place to produce reliable information about student perform- 



■ Are local entities in a state using similar measures and assessing outcomes for the 
same population of students? 

■ Have challenging, yet realistic, standards of performance been established? 

■ Why have some states made greater progress in developing accountability systems 
for vocational education than others? 

■ To what extent are states and local entities able to integrate WIA and Perkins per- 
formance accountability requirements? 

^ How are Perkins performance report results used to manage and improve programs? 

In addition to highlighting data quality concerns, the earlier accountability studies of Perkins III 
also questioned the utility of the data collected. Local educators reported seeing little value in the 
information they were asked to gather and report. They therefore failed to use the data as intended 
for program improvement, providing even less incentive for them to ensure high data quality. 
Given this history, it will be important to examine several questions about the impact of the new 
requirements: 



‘‘‘States can renegotiate levels of performance prior to the third program year covered by the state plan. Timelines for 
the NAVE report will allow it to describe any changes in standards that may occur, but not the resulting outcomes. 
‘^Mikala Rahn and Patricia Holmes, “Accountability Systems: Performance Standards and Assessment,” NCRVE Cen- 
terpoint (Match 1999). 
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■ Are states and communities using the data gathered to manage and improve their 
programs (i.e., are one-stop clients given access to this information)? Have account- 
ability requirements resulted in any unintended effects? 

■ Are the incentive payment provisions credible? 

■ Can state reported performance data be aggregated to produce a national profile? 
Can it be used to compare state performance? 



Planned Studies 

NAVE will examine accountability practices at both secondary and postsecondary levels 
through: 

■ Surveys of state directors. 

■ Review of state reported performance data. 

■ In-depth case studies of state and local accountability practices. 



B. FUNDING ISSUES 

It has been frequently observed that federal policy is a “blunt instrument” for change. However, 
nowhere is the impact of federal legislation more direct and immediate than the rules governing 
allocation of funds. The Congress that enacted Perkins III was guided by two major principles 
regarding funds (1) “drive” more money to local grantees, and (2) provide greater flexibility re- 
garding the use of funds. 




ERIC 
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Toward these goals, the Act (1) eliminated a 10.5 percent set-aside for single parents and dis- 
placed homemakers, enabling Congress to direct that 85 percent (up from 75 percent) of funds 
should go to local programs; (2) created a 10 percent reserve fund which may be distributed out- 
side of the intra-state formula; (3) eliminated the requirement for a state sex equity coordinator 
while mandating that $60-150,000 of state leadership funds be used for sex equity services; (4) 
eliminated rules regarding prioritizing local funds to serve programs with the highest concentra- 
tion of special populations; and (5) changed the secondary school intra-state formula to simplify 
but not significantly alter funding allocations. Also relevant was the elimination of an 8 percent 
set-aside for school programs as Congress replaced the Job Training Partnership Act with WIA. 
NAVE will examine both the allocation and use of funds under the new Perkins provisions, in- 
cluding; 

To what extent have the new provisions affected how funds are allocated? 

■ How have changes in the intra-state formula, and reserve fund, altered the amount 
and distribution of funds at the local level? 

■ Have states changed the allocation of funds between secondary and postsecondary 
education? 



•* How has the use of funds changed, if at all? 

■ What has been the effect of eliminating set-asides and other provisions for special 

■ populations? 

■ How do states use “state leadership” funds? 

■ What priorities are emphasized by local education agencies and postsecondary in- 
stitutions in their use of Perkins funds? To what extent does the use of funds support 
priorities in Perkins HI? 



Planned Studies 

Surveys of state directors (secondary and postsecondary) 
In-depth case studies of state and local priorities. 



NAVE Overview— 23 



OTHER MANDATED ISSUES 



Congress mandated an assessment covering a broad range of issues. The preceding discussion 
addressed a general approach to nearly all of the topics in the congressional mandate — as well as 
some others. Two topics not addressed, however, are (1) teacher supply and demand, and (2) the 
role of technology and distance learning in vocational education. The current plan for examining 
these other key issues include: 

Teacher Supply and Demand 

NAVE plans to utilize findings from a study of the supply and demand for vocational education 
teachers that is currently being conducted by NCRVE. Complementing this research will be in- 
formation gathered from the comparative case studies (described previously). 

Technology and Distance Learning 

Several efforts are currently underway to track the growth of distance learning programs of study, 
and to understand the instructional role of technology. NAVE plans to summarize this research as 
it relates to occupational training. How vocational curriculum responds to changes in commercial 
technology is a topic to be explored in the case studies. 
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